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(57) Abstract: A method for simulating an appearance of make-up and fashion accessories on an image of a consumer and for 
marketing such products includes compiling a database of a plurality of products having appearance information for each product, 
acquiring a base image of a consumer having a plurality of pixels, identifying a product application area in the base image having 
a blending region, where the product application area and blending region thereof being are by sets of pixels of said base image, 
receiving a product selection from a consumer, retrieving appearance information associated with a selected product from the data- 
base, modifying the appearance information of the product application area according to the appearance information of the selected 
product, blending the blending region with the base image by modifying appearance information of pixels of the blending region 
according to appearance information of the selected product and according to appearance information of associated pixels of the base 
image in the blending region, displaying a composite image of the base image as modified and blended in the application area and 
the blending region thereof, and providing means to display an alternative product within the product application area. 
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MAKE-UP AND FASHION ACCESSORY DISPLAY AND MARKETING 
SYSTEM AND METHOD 

Field of the Invention 

The invention pertains to the field of display systems and methods for simulating the 
5 appearance of make-up and fashion accessories, and pertains to a method for marketing make-up 
and fashion accessories. 

Background and Summary of the Invention 

Consumers of make-iq) and fashion accessory products, especially "high-end" make-up 
and fashion accessory products, usually need to see how such products will appear on themselves 
10 prior to making a purchase decision Because of this, make-up and fashion accessory merchants 
have had aneed to rely substantially on physical store locations where consumers can tiy different 
make-up applications and accessories. 

Some merchants have attended to use digital computer imaging technology to provide a 
simulation of make-up products, however, these past systems merely display various types of 
15 make-up on models and do not show how such make-up would appear on the consumer. 
Moreover, these past systems use fixed photographs and do not provide means to select various 
combinations of products. Further, the past systems do not provide means to adjust the amount 
of make-up products applied 

Therefore, what is desired is a system and method for life-like simulation of the 
20 appearance of make-up and fashion accessories on an image of a consumer, which allows the 
consumer to simulate various combinations of products and allows the consumer to simulate 
various application amounts of make-up products. Further, what is desired is a method of using 
an image of a consumer as a vehicle for marketing make-up products and fashion accessories. 

The present invention provides a system and method in which make-up and accessory 
25 products precisely overlie and smoothly blend with a digital image of a consumer (e.g., an image 
of the consumer's face, or head and torso) to create a true-to-life simulation of how such products 
will appear on the consumer. Further, the system and method allows the consumer to choose 
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from a large plurality of products, to create a multiplicity of combinations of image simulations, as 
desired by the consumer. Further, the system and method allow the user to modify the image to 
simulate applications of various amounts of make-up products, as preferred by the consumer. In 
addition, the invention provides a method for marketing make-up products and fashion accessories 
5 to a targeted consumer using a digital image of the consumer, where products precisely overlie 
and smoothly blend with the digital image, to create a compelling and effective marketing vehicle. 

Brief Description of the Drawings 

For a complete understanding of the above and other features of the invention, reference 
shall be made to the following detailed description of the preferred embodiments of the invention 
10 and to the accompanying drawings, wherein: 

FIG. 1 is a schematic overview of the components of a system according to the invention 

FIG. 2 is a screen shot of an initial interface of the system 

FIG. 3 is a screen shot of an image simulation interface 

FIG. 4 is a screen shot of a retrieval interface for loading stored looks 

1 5 FIG. 5 is a process flow diagram of the method of the invention 

FIG. 6 is a color image of a base image of a consumer 

FIG. 7-11 are color composite images showing make-up applied to various make-up 
application areas of the base image 

FIG. 12-18 are gray-scale images representing alpha channel values of several make-up 
20 application areas of the base image 

FIG. 19 is a graph of the equalization function of the present invention; 

FIG. 20 is a graph of the distribution of the slope A of the equalization algorithm as a 
function of the intensity of the product color; and 
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FIG. 21-25 are composite images of various fashion accessories superposed over the 
base image 

The file of this patent contains at least one drawing executed in color. Copies of this 
patent, with color drawings, will be provided by the Patent and Trademark Office upon request 
5 and payment of the necessary fee. 

Detailed Description of the Invention 

Referring to FIG. 1 , the method and system of the present invention provides a means for 
consumers to "try and buy 0 make-up products and accessories, from virtually any computer in 
virtually any locatioa The components of the system 10 include a host system 12 and a plurality 
10 of user systems 14, 16 interconnected by a computer network 4* 19. Preferably, the components 
of the system 10 are interconnected by the Internet such that the user systems 14, 16 can be 
located anywhere in the world with access to the Internet. 

The host system 12 includes a host computer 18, which can comprise one or more 
computers, and a database 20. The user systems 14, 16 each include a graphic monitor 22 and a 
15 central processing unit 24, which can be of any suitable type. The user systems 14, 16 can be 
"thin" clients, with minimal resources, or can have substantial resources including a data storage 
device, such as a hard drive. 

As depicted, the components of the system 10 can be distributed over a wide geographical 
area. However, it can be appreciated that some or all of the components can be included in one 
20 enclosure. For exanyle, it is contemplated that all of the components of the system can be 
enclosed within one "kiosk", which could be located within a retail store. 

The system uses a digital base image of the head and face of the user to demonstrate how 
make-up and accessory products will appear on the user. This base image can be considered a 
blank canvas of the user and can be acquired using an image capturing device 26 connected to the 
25 usersysteml4, 16 or can be acquired by digitally scanning a photograph 28 sent to the hostl2by 
the user. Other methods of acquiring the digital base image are also suitable. 



WO 02/05249 



4 



PCT/TO01/01495 



The database 20 includes product information for various types of make-up products and 
fashion accessories, including appearance information to be used in simulating the appearance of 
the products on the base image of the user. The method requires only a very "thin" amount of 
data content to be stored in the database for each product - a product's color is represented by a 3 
5 byte RGB value, which is enough for blending engine to produce the photo-realistic look. Two 
major advantages of this method include the ability to transfer product information quickly over a 
very narrow-bandwidth communication line (Le., low transmission overhead) and the 
compatibilityof the product information with many types of user systems 

The client application is built around the color blending and display software engine. The 
10 client application has a protocol for querying the database and getting products details, color, and 
accessories images. It also has a user interface through which the user controls the application of 
colors and accessories to the user's image. These functional controls can be hooked to any menu 
system, keyboard, remote control or graphic user interface. The description below describes the 
system as implemented via the Internet graphic global computer network. However, it can be 
1 5 appreciated that other implementations are within the scope of the inventioa 

The System User Interfaces 

FIG. 2 displays an initial graphic user interface 30 of the system, suitable for access over 
the Internet through a standard browser. The initial interface 30 includes means for new users to 
register with the host system 12 and means for registered users to login (using a unique user-name 
20 and password combination). 

Referring to FIG. 3, an image simulation graphic interface (ISI) 33, for the internet, 
accessible through a standard Internet browser, provides a single interface through which users 
can "try and buy" cosmetic make-up products and fashion accessories, and other related products. 
The ISI 33 includes the user's base image 02, a category list 32, and a subcategory list 34. 

25 The categories in the category list 32 can include eyes, lips & face, accessories, nails, hair, 

and other categories of related items. The subcategory list 34 is dependent upon the category 
chosen (Le., highlighted or clicked upon). As shown, if the lips & face categoiy is chosen, the 
subcategory list 34 can include: lipstick, lip-gloss, lip pencil, blush, foundation, power, and others. 
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Subcategories for the lips categoiy can include: eye shadow, eyeliner, mascara, eye pencil, etc. 
Subcategories for accessories can include: eye wear, earrings, necklaces, pendant, and the like. 
The subcategories for nails can include various types of nail polish. Similarly, the subcategories 
for hair care can include various hair dies and highlights, and the like. 

s The ISI 33 also includes a brand name list 38, which is retrieved from the database 20 and 

which is preferably dependent upon the categoiy and subcategory chosen by the user. The ISI 33 
also includes a products list 36, which is preferably dependent upon the categoiy, subcategory and 
brand name chosen by the user. 

The ISI 33 also includes a color selection interface (CSI) 44, which displays color options 
10 available for the chosen product Preferably the CSI 44 displays relatively enlarged graphic 
images of the products to cleariy indicate the various color options. The user can chose a color by 
highlighting or clicking upon one of the colors displayed in the CIS 44. Upon such a selection, 
the system 10 displays the chosen product on the base image 02 of the user in the ISI 33 in a 
method describe in detail below. When a color is chosen from the CSI 44, the system 10 also 
15 preferably displays a full image of the product and its packaging in a product details section 40 of 
the ISI 33, along with descriptive and price informatioa 

Referring to FTGs. 6, the base image 02 of the consumer is preferably an image of the 
consumer without any make-up (or with only base make-up); thereby forming the "blank canvas" 
upon which make-up products will be applied, with the digital blending technologies of the 
20 invention. 

FIGs. 7-10 display how the base image is altered by the system when make-up products 
are applied to various portions of a base image. Specifically, FIG. 7 shows red blush applied to 
the cheeks on the base image; FIG. 8 shows blue eye liner applied to the upper eye lid on the base 
image; FIG. 9 shows red lipstick applied to the lips on the base image; and FIG 10 shows purple 
25 eye shadow applied to the upper eye lid on the base image. 

The ISI 33 also includes application controls 45 to adjust the "amount 11 of the make-up 
product applied to the base image 02. Specifically, the application controls 45 include means to 
increase the amount of make-up (Le., "heavier 1 '), to decrease the amount of make-up ("lighter") 
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and to reset the application amount to an initial, or defeult amount ("reset"). 

FIG. 11 is image of a significantly reduced amount of eye shadow applied to the base 
image as compared to that of FIG. 10. 

The ISI 33 also includes a save function 54 which directs the system 10 to save 
5 information regarding the current state of the image in the ISI 33, including the products currently 
applied to the image and any variations in the application of the products directed by the user 
through the application controls 44. For purposes herein, such information will be collectively 
(ailed a "look" 

The look information is preferably recorded in a specified format, either on the user's 
10 system 14,1 6 or on the host 12, or both. Preferably, the look file also contains a look image file 
representing the base image with the products identified in the lo ok file applied thereto. The look 
image can be of a reduce size and/or resolution. The system 10 is able to use the product 
information contained in a look file with any base image of any user. 

Referring to FIG. 4, the system 10 includes a retrieval interfece 72 for retrieving saved 
15 "looks. Preferably, the retrieval interface 72 includes a relatively small (e.g., thumbnail) image of 
one or more saved looks such that the user can conveniently and efficiently identify a desired look 
to load. Also, upon initial selection or highlighting of a saved look (e.g., by clicking once upon 
the associated thumbnail image), the retrieval interface 72 preferably displays written details 
regarding the products included in the look in a product detail portion 70. When a look is chosen 
20 from the retrieval interface 72, the system changes to the image simulation interfece 33 and 
displays the chosen look, whereby the user can manipulate the look as previously described. 

The ISI 33 also includes an email control 58 to send the current look information and/or 
current image to another as an attachment to an electronic mail message. The system 1 0 provides 
means to allow a recipient of such a message to load the image created by the sender (if supplied) 
25 or to apply the look to their base image 02. 

When appropriate, the ISI 33 also includes controls to adjust the appearance of 
accessories displayed with the base image (not shown). For example, the ISI 33 preferabfy 



WO 02/05249 



7 



PCT/IB01/01495 



includes controls to adjust the scale and orientation of the image of the accessory. Further, the 
ISI 33 includes controls to clear 60 all product from the base image 02, and to print 56 the current 
image on a printer. 

Further, the ISI 33 preferably includes an electronic "shopping cart" section including 
5 controls to add 46 and clear 48 products to and from the shopping cart, and a control to buy 50 
products currently in the shopping cart. Thus, the image simulation interface 33 provides a 
means for consumers to select various make-up and accessory products, visualize how those 
products will appear on themselves, and then purchase those products, all from one interface. 

The Image Simulation Process 

10 Referring to FIG. 5, an initial step 1000 of the method of the invention is compiling a 

database of information regarding a plurality of make-up and fashion accessory products. For 
make-up products, the database includes the product name, brand, category, subcategory cost, 
and a brief description of the product, . The database also includes appearance information for a 
make-up products includes a three-bit RGB color representation and two 1 byte parameters for 

15 layer texture and light reflectivity (i.e., "glossiness"). For accessory products, the database 
includes the product name, category and cost, one or more digital images of each product, and an 
anchor point for each digital image. 

The method also includes a step 2000 of acquiring a digital base image of a user. As 
stated above, this can be accomplished through any suitable digital image-capturing device or 
20 system, and can be made by imaging the user, or by digitally scanning a photograph of the user. 

The base image is then processed 3000 to identify areas in which make-up and accessory 
products are to be applied This processing is preferably performed at the location of the host 
system 12 and can be done by any suitable combination of manual and automated processing. 
Pixel information of the base image is preferably stored in a base image file, in a bitmap RGB 
25 bands in, for example, a "tiff * format. The information regarding placement of make-up products 
on the base image, is also stored in the base image file along with the base image information, in 
separate "alpha channels' 9 , in the tiff file. 
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The first process in step 3000 is determining the base image size factor and rotation factor 
according to the size and orientation of the head/face of the user within the base image. 
Specifically, a pixel dimension of the head/face in the base image is compared to (Le., divided by) 
a predefined standard dimension to calculate the size factor. A primary rotation factor (i.e. a z- 
5 axis rotation factor) is calculated by comparing an axis of the base image (e.g., the longitudinal 
axis of the face) with the vertical axis of the base image. Secondary rotation factors (i.e., y-axis 
rotation factors) can also be computed for other features of the base image, such as the orientation 
of the ear lobes in the base image for property orienting drop earrings. The image can then be 
cropped to the computed size and rotation. 

10 

Further, anchor points for locating accessoiy product images are defined by identifying 
(x,y) coordinates of associated pixels within the base image. For example: a point midway 
between the pupils can be identified as an anchor point for eye wear; points adjacent each ear lobe 
can be identified as an anchor points for earrings; and the two points where a necklace would 
15 disappear (as it passes around the neck) can be identified as a anchor points for necklaces. 
Further, for eyewear, the system calculates a rotation angle between a line connecting the pupils 
and the axis of the base image. The anchor points, along with the size and rotation factors for 
eyewear are recorded within the base image file, preferabfy in a header portion thereof. 

For each base image , one or more make-up application areas are defined for each make- 
20 up product type and are assigned a separate alpha channel within the base image file. Each alpha 
channel is a pixel-by-pixel map representing where and how much of an associated make-up 
product should be placed within the base image. Preferably, the alpha values in the alpha channels 
range in scale from an upper limit (e.g., 255) to a lower limit (e.g., 0). An alpha value equal to the 
lower limit (0) indicates that the color of the associated make-up product should be applied to the 
25 associated pixel in the base image, without modification (Le., without blending). An alpha value 
equal to the upper limit (255) indicates that the color of the make-up associated make-up product 
should not be applied to the associated pixel in the base image. 



The make-up application areas have blending regions wherein the alpha values range 
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between the upper and lower limits (0<Alpha<255) indicating that the color of the associated 
make-up product should be blended, according to the alpha value, with the color of the associated 
pixel of the base image. 

Referring to FIGs. 12-1 8, for explanatory purposes, the alpha values for various make-up 
application areas are depicted in gray-scale images reflecting the alpha values for associated pixels 
in the base image. In the representation employed in FIGs. 12-1 8, a darker shade of gray indicates 
a low alpha value, which results in a greater weighting given to the color of the make-up product 
during the blending process. Conversely, a lighter shade of gray indicates a high alpha value, 
which results in a lesser weighting given to the color of the make-up product. - 

FIG. 12 displays alpha values suitable for blending lip liner products, on the periphery of 
the lips. FIG. 1 3 displays alpha values suitable for blending lipstick products on the lips. FIG. 14, 
displays alpha values suitable for blending blush products on the cheeks. FIG. 15 displays alpha 
values suitable for blending eyeliner products on the upper eyelid. FIG. 16 displays alpha values 
suitable for blending eyeliner products on the lower eyelid FIG. 17 displays alpha values suitable 
for blending eye shadow on a lower portion of the upper eyelid And, FIG. 18 displays alpha 
values suitable for blending eye shadow on an upper portion of the upper eyelid. 

At the end of the processing line, the data size of the base image file is preferably reduced 
Specifically, the three bands of the image are compressed and the alpha channels are encoded in a 
unique format with which the color-applying engine works. 

The base image file, containing the information obtained during processing, can be stored 
on the host system 1 2 and transmitted to the client system 14, 16 as needed (at session startup for 
instance). Alternatively, if the client system 14, 16 includes a storage device, the base image file 
can be transferred to the client system 14, 16 in any suitable manner - for example by transmitting 
the c base image file over the network 19, or by recording the base image file on a portable storage 
medium and sending the medium to the user through the mail- and then recorded on the user 
system For Internet based implementations, the user can download the processed base image 
from the host 12 over the network 1 9, or can receive the base image via electronic mail 

Referring again to FIG. 5, in step 4000A, the user selects a product from the database 20 
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using the image simulation interface (ISI) 33, as described above. In response to the user's 
selection, the host system 1 2 sends the product information stored in the database 20 to the client 
system 14, 1 6. To simulate the appearance of the selected product on the base image of the user, 
the color information of the selected product is modified and blended according to the color 
5 information of the base images in steps 5000 and 6000, to create a composite image. 

In the modification step 5000, the appearance information of the base image (or at least 
the appearance information of the associated make-up application area) is modified, simulating the 
physical process of color application, to create the appearance that the make up region has been 
fully covered with the color of the product, and was photographed at the same illumination and 
10 shading as the base image. This is achieved by using the intensities of the make-up area of base 
image (as a black and white image) to which the product color is applied. 

First, the image is converted from the Red-Green-Blue (RGB) format to aHue-Saturation- 
Intensity (HSI) format. Then, the intensity image undergoes a contrast stretching (or 
"equalization*') procedure in which the intensity values (I) of the pixels in the make-up application 

15 area are adjusted according to a predetermined equalization algorithm. The arguments of the 
equalization algorithm are: (1) the difference in intensity between the applied color and mean 
intensity of the pixels of the make-up area, and (2) the variance of the intensities of the pixels of 
the make-up area. The hue and saturation values (H,S) of the pixels in the make-up application 
area are set equal to the hue and saturation values of the selected make-up product. This process 

20 preserves details, highlights and shadows present in the original base image. 

Referring to FIGs. 1 9 and 20, the algorithm for equalizing the intensity values of make-up 
application area is a linear function whose parameters are computed from the statistics of the base 
image make-up application area and the intensity of the product color, as follows: 

lout = A * Iin +B 

25 Where; 

lout is the value of the intensity of a pixel of the equalized area , 



lout can range between Upper and Lower Limits of 255 and 
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0, respectively, 

lb is the value of the intensity of an associated pixel of the make- 
up application area of the base image (before equalization), 

Iin can range between the Upper and Lower Limits, 

5 A is the slope of the function and is computed according to a 

gaussian distribution function, as follows: 

A « exp(-0.5 * (Icolo^Imcj'/Ulcv 2 ) 

Where, 



10 



Icoior is the value of the intensity of the applied 
product color, 

Inean is mean value of the intensities of the pixels in 
the make-up application area, and 

I*dev is the variance of the intensities of the pixels in 
the make-up application area, 



15 B is a constant computed by equating the 1^ and 1^ values in the 

equalization function to Imean and Icoior, respectively 

According the above formula, pixels in the make-up application area of the base image 
having the mean intensity value of the make-up application area will be assigned an intensity value 
exactly equal to the intensity value of the product color. Pixels of the make-up application area of 
20 the base image having intensity values other than the mean intensity value will be assigned 
intensity values spread around intensity value of the product color according to the variance in 
intensity of the make-up application area of the base image. 

For extreme differences in intensity value between the applied product color and the pixels 
of the make-up application area of the base image, the slope A of the equalization function is 
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modified according to the following: 

For Icolor - Iiaem > 2* Istdev, 

Imcan is replaced by W0.4 + Icobr* 06 
For Icoior - Imean < -2*l&dcv, 
5 Imean IS replaced by 1.2 * I mC an 

The equalization function can be represented in one or more look-up tables stored on the 
client system and/or the host system, as appropriata 

After applying the intensity equalization function and changing the hue and saturation to 
the associated values of the applied color in the make-up application area, next, the H,S,I values 
10 of the pixels in the make-up application area are re-converted back into the RGB format, by a re- 
conversion procedure, using an inverse of the transformation described above. 

In a blending step 6000, the pixels of the blending region of the make-up application area, 
which by definition have alpha values between the upper (255) and lower (0) limits, are blended 
with the associated pixels in the original base image to form the fully colored base image. An 
15 oflset value (constant to all pixels in the make-up application area) is also introduced which 
governs the "amount" of applied color relative to the amount of original color in final image. This 
oflset value is controlled from the 'Heavier'VXighter" color application controls on the image 
simulation interface (ISI). 

Specifically, the red, green and blue color values (Rc, Gc, Be) of the pixels of the blending 
20 region of the make-up application area are determined by the following formulae: 

Rc= [Rb * Alpha + Rp * (Alpha - Upperlimit)]/Upperlimit 

Gc= [Gb * Alpha + Gp * (Alpha - UpperLimit)]/UpperLimit 

Bc= [Bb * Alpha + Bp * (Alpha - UpperLimit)]/TJpperIin± 



Where: 
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Rb, Gb, Bb are respectively the red, green and blue color values of the associated 
pixels in the original base image; 

Rp, Gp, Bp are respectively the red, green and blue color values of the associated 
pixels in the blending region of the modified make-up application area; 

5 Alpha is the Alpha value of the associated pixel in the blending region of the 

make-up application area minus the alpha oflset value obtained from the 
"heavierTTighter" controls; and 

UpperLimit is the predefined upper limit of the Alpha values. 

Upon completion of the modification and blending steps 5000, 6000, the composite image 
10 is displayed 7000A in the image simulation interface (TSI) 33. It should be noted that the 
calculations of die above modification and blending steps could be made by the user system 14, 
16, by the host system 12, or by any combination thereof. 

The combination of defining accurate make-up application areas and blending regions (in 
15 the processing step), preserving details, highlights and shadows (in the modification step), and 
blending colors from the base image in the blending region of the make-up application area On the 
blending step) achieves a very high, and heretofore unavailable level of realism for simulating 
make-up applicatioa 

In a further adjusting step 8000, the user can adjust the Aamount@ of make-up applied by 
20 the system to the base image. As set forth above, the ISI 33 includes application controls 45 to 
increase or decrease the amount of make-up applied to the base image. These controls effectively 
adjust the offset to the alpha values of the pixels of the blending region to affect the respective 
weights given to the colors of the pixels of the original base image and of the pixels of the 
blending region of the modified make-up application area 

25 Referring to, when the user directs the system to lighten the application of make-up 

(compare FIGs. 1 0 and 1 1), the alpha values for the blending region of the make-up application 
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area are increased by a predetermined amount to increase the weight given to the colors of the 
pixels of the original base image. Then, the blending step 6000 is repeated to recalculate the 
blended colors to create an adjusted composite image, and the adjusted composite image is re- 
displayed in the ISI 33. 

5 Conversely, when the user directs the system to increase the application of make-up, the 

alpha values for the blending region of the make-up application area are decreased by a 
predetermined amount to decrease the weight given to the colors of the pixels of the original base 
image. FIG. 10 shows a make-up product applied in a standard or default amount. FIG. 1 1 
shows the same make-up product applied in a lighter amount. Preferably, the system imposes a 
10 limit upon how much the user can increase the application of a given make-up product so as to 
avoid undesirable results. For example, the system can limit the increase in application by 
between about 40% and 100% of the standard or default amount. 

Make-up products are often applied in completely or partially overlapping layers. For 
example, blush is often applied over a foundation, and different shades of eye shadow make-up are 
15 commonly applied in overlapping layers. Depending on the amount of application and color of the 
overlying product, the underlying product will often affect the appearance of the overlying 
product. Therefore, in rendering images, which include such overlapping make-up products, the 
system blends the colors of the products together in the region in which they overlap. 

To render the image of overlapping make-up products, the system computes the 
20 composite image in iterative, sequential steps. First, the system computes the composite image 
with the underlying make-up product, in the manner set forth above (i.e., modifying and blending 
with the original base image). Then, the system applies the overlying product in the same maimer 
using the first composite image as the "base" image. The system applies overlapping make-up 
products in a predetermined order, generally recognized as the proper order of application. 

25 Referring to FIG. 3, the ISI 33 includes controls 47, 48 to apply eye shadow to either a 

primary area or a secondary area, which overlap, as can be seen from the graphic representations 
of the areas adjacent the controls. Preferably the controls 47, 48 function similarly to so-called 
"radio buttons", and are mutually exclusive, whereby turning one "on" turns the other "off." 
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Faghipn Accesspry Product? 

To render composite images of accessory products with the base image, the system aligns 
the anchor point of the image of the accessory product (which can be the first pixel in the image 
or any other predefined pixel) with the associated anchor point coordinate contained in the base 
5 image file (described above). The system adjusts the size of the image of the accessory product 
according to the size factor contained in the base image file. Further, the system adjusts the 
orientation (angle) of the accessory image according to the rotation factors) contained in the base 
image file. 

FIGs. 21 and 22 display accessory products rendered with the base image. 
10 Specifically, FIG. 21 shows drop earring accessory products correctly aligned so as to appear to 
be hanging from the ear lobes of the consumer's image. As can be appreciated, the orientation of 
each earring is adjusted to appear realistic. Specifically, each earring accessory products have 
been separately rotated around a y-axis so as to appear to hang naturally. FIG. 22 shows a hat 
superimposed over the base image. 

15 Referring to FIG. 23, the product image may include transparent areas (e.g., the center 

area of an image of a loop earring, or the area surrounding the product in the image) in which the 
base image will appear unmodified. The product image may also include semi-transparent areas 
(e.g. tinted lenses of eye wear as shown in FIG. 23) in which the product image is blended with 
the base image. Thus, after alignment, size adjustment and orientation of the accessory product 

20 image, the system blends the product image with the base image using the alpha values of the 
pixels of the product image. 

In a processing step similar to that described above for make-up application, the semi- 
transparent areas of the product image are assigned alpha values between the upper and lower 
limits (255 9 0) t thus creating blending regions in the product image. As stated above, the alpha 
25 values are preferably contained in alpha channels in the product image file. 

In a blending step similar to that described above, the color information of the pixels in the 
blending regions is blended with the color information of the pixels of the base image, using the 
blending algorithms set forth above. Upon completion of the blending step, the composite image 
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including the blended product image is displayed in (he ISI 33. 

To render images of hair products, the system employs image processing, modification and 
rendering steps similar to those described above. Specifically a hair product application area is 
defined, with associated alpha values and the color of the hair in the base image is changed using 
the modification and blending steps as above. As above, the system preferably allows the user to 
adjust die color effect of the hair product. The information regarding applied hair products can be 
saved in (and loaded from) the product information "look" file, as described above. 

Referring to FlGs. 24 and 25, to render images of nail products, the system preferably uses 
a "stock" image of a hand (FIG. 24), from for example a hand model, and similar steps are used to 
simulate nail products (e.g., nail polish) applied to the nails of the image. Thus, nail product 
application areas are defined (i.e., the nail areas), with associated alpha values (if necessary), and 
the color of the nails in the image is changed using the modification and blending steps as above 
(FIG. 25). As above, the nail product information can be saved in a product information file. 

The Marketing Methods of fee Invention 

The marketing method of the invention provides new and powerful tools for promoting 
cosmetic and fashion products, most suitably via the Internet. Retailers can use the technology of 
the invention to promote various products and introduce to their potential clients a dependable 
simulation 

of how they will look wearing their products. Using customer's own photos, and based on 
shopping behavior, retailers and manufacturers can direct their marketing efforts to a focused 
audience and feature a new and vivid experience. 

In one marketing method of the invention, the system 10, selects one or (preferably) more 
make-up, hair, nail and/or accessory products, or other products from the database 20. This 
selection can be made at the request of a third party (e.g., a manufacturer of the products) without 
direct input from the consumer. The system then creates either a fully rendered composite image 
of the consumer with the selected products, using the consumer 1 s own image as the base image, or 
merely creates a "look" information file containing the selected product information. Next, the 
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system sends the fully rendered composite image, or the look information file to the consumer. 

The fully rendered composite image or the look information file can be delivered to the 
consumer in any suitable manner including electronically (e.g., via electronic mail), in a 
photograph, or on a computer-readable medium (e.g., on a CD). In addition, samples of the 
5 products shown in the composite image or file can be delivered to the consumer simultaneously. 

If the composite image or look information file is delivered to the consumer electronically, 
or in a computer-readable format, the consumer can use the client system 1 4, 1 6 to load the image 
or file through the retrieval interface, and can view the resulting image through the ISI 33. 

In another marketing method of the invention, 'look codes" are posted or published, for 
10 example in traditional advertisements (e.g. print, television or radio ads) or electronic 
advertisements (e.g, banner ads in web pages), which codes are associated in the system with 
predetermined make-up and/or accessory products, and particular applications of such products 
(i.e., various make-up 'looks"). Preferably such codes appear in advertisements of manufacturers, 
which advertisements depict or describe make-up products applied to models, and the codes are 
15 aligned near (or otherwise associated with) a brand name of the system (e.g., EZFace). Upon 
access to the system, the consumer enters the "look code", which directs the client and/or host 
system to display the products over the (previously captured and processed) base image of the 
consumer, as described above, so that the customers can see for themselves how the products 
will look on themselves. This method enables manufacturers to efficiently demonstrate their 
20 products to customers in a discrete and personalized manner. 

In another marketing method of the invention, Internet web pages are provided with 
electronic links (i.e. hypertext links), which when clicked upon, cause the client system to display 
the consumer's image with a predetermined application of make-up products and/or accessories. 
The links can take any suitable form (ie., text links or image links). Preferably, the links are 
25 aligned adjacent to or otherwise associated with images of models wearing the make-up 
applications, or associated with make-up or accessory products. 

If the user interfaces of the system are implemented through a standard browser, such as 
with a "plug-in" or script, then the link can include a network address of the host system 12 (i.e., 
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the url) and an embedded look code which is received by the host system 12 upon clicking upon 
the link. Upon login to the system, the system then uses the embedded look code to retrieve the 
look information from the product database, and thai displays the products on the image of the 
consumer, as described above. 

5 If the user interfaces are implemented through a separate client application, the client 

system can include a listening program running in the background which monitors when a 
hypertext link is clicked upon The listening program retrieves the embedded look code, launches 
the client application (if required), and passes the look code to the client application, when then 
contacts the product database (through the network and host) to retrieve the product information, 

10 and to display the composite image. 

Further, the system can record a consumer's buying and "trying" habits in a consumer 
histoiy database located on the host system 12, which can then be used to target personalized 
product marketing to the consumer, or to conduct highly accurate market studies about consumer 
preferences. 

15 It can be appreciated that the marketing method of the invention provides a powerful and 

persuasive marketing tool because the user can see the products as applied to their own image, 
and, through the system can easily and conveniently purchase the products. 

Multi-User Sessions 

In addition, the system 10 provides means whereby two or more users can manipulate a 
20 single base image simultaneously. This provides a powerful tool whereby friends or students can 
try different make-up applications together, in an effective and entertaining manner. 

In this method, a first client system 14 and a second client system 16 are interconnected 
via the host system 12, through the network 19. Hie initial interface 30 of the system 10 
preferably includes a link, labeled for example "Shop with a Friend", which when selected 
25 preferably provides means for each user to select another user or group. When this feature is 
enabled, the ISI 33 of each user displays an identical base image, which can be the base image for 
one of the users. Modifications made at one client system 14 are monitored by (transmitted to) 
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the host system 12, and then transmitted to the other client system 16, where the modifications are 
made or displayed on the ISI 33 of the receiving client system, in accordance with the methods 
described above. 

Optionally, the system 1 0 prohibits one of the users from making modifications to the base 
image. Similarly, if a multiplicity of users are similarly interconnected through the system (e.g., in 
a group or class), the system prohibits all but one of the users (e.g., the instructor) from making 
modifications. The information flow and routing of information over the network 19 during 
multiple-user sessions can be managed by the host system 12 or a dedicated system for such 
function, which can monitor user requests and can log pre-scheduled dates for 4< net meetings." 

It should be understood, of course, that the specific form of the invention herein illustrated 
and described is intended to be representative only, as certain changes may be made therein with- 
out departing from the clear teachings of the disclosure. 



WO 02/05249 



20 



PCT/IB01/01495 



What is claimed: 

l.A method for simulating an appearance of products on an image of a consumer, comprising 

- compiling a database of a plurality of products, said database including appearance information 
for each said product 

- acquiring a base image of a consumer, said base image including a plurality of pixels 

- identifying a product application area in said base image, said product application area including 
a blending region, said product application area and said blending region thereof being defined by 
sets of pixels of said base image 

- receiving a product selection from said consumer 

- retrieving appearance information associated with said selected product from said database 

- modifying appearance information of said pixels of said product application area according to 
said appearance information of said selected product 

- blending said blending region with said base image by modifying appearance information of said 
pixels of said blending region according to appearance information of said selected product and 
according to appearance information of associated pixels of said base image in said blending 
region 

- displaying a composite image of said base image as modified and blended in said application area 
and said blending region thereof 

- providing means to display an alternative product within said product application area 
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2. A method for simulating an appearance of products on an image of a consumer, as in claim 1, 
further comprising: 

5 - assigning alpha channel values to each pixel of said set of pixels of said product application area, 

- blending said product application area with said base image by modifying appearance 
characteristics of pixels of said product application area according to appearance information of 
said selected product and according to appearance information of associated pixels of said base 
10 image 

in said blending step, weighing said appearance information of said associated 
pixels of said base image and appearance information of said selected product according 
to alpha valuesofassociatedpixelsofeaidmake-upapplicationarea 

15 3. A method for simulating an appearance of products on an image of a consumer, as in 

claim 2, further comprising: 

-in said blending step, weighing said appearance information of said associated pixels of said base 
image in reverse proportion to appearance information of said selected product 

20 

4. A method for simulating an appearance of products on an image of a consumer, as in claim 3, 
further comprising: 

a red color value (Rc) of said pixels of said product application area in said composite 
25 image being determined by the following formula- 
Re^ [Rb * Alpha + Rp * (Alpha - UpperLimit)]/UpperLiniit, where 
Rb is a red color value of an associated pixel in said base image 
Rp is a red color value of said selected product; 
30 Alpha is an Alpha value of an associated pixel in said product application area; 

and 
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UpperLimit is a predefined upper limit of said Alpha values 

a green color value (Gc) of said pixels of said product application area in said composite 
image being determined by the following formula; 

5 

Gc= [Gb * Alpha + Gp * (Alpha - UpperLimft)]/UpperIimit, where 
Gb is a green color value of an associated pixel in said base image 
Gp is a green color value of said selected product; 

Alpha is an Alpha value of an associated pixel in said product application area; 

io and 

UpperLimit is a predefined upper limit of said Alpha values; 

and 

a blue color value (Be) of said pixels of said product application area in said composite 
15 image being determined by the following formula: 

Bc= [Bb * Alpha + Bp * (Alpha - UpperLimft)]/UpperLimit, where 
Bb is a blue color value of an associated pixel in said base image 
Bp is a blue color value of said selected product; 
20 Alpha is an Alpha value of an associated pixel in said product application area; 

and 

UpperLimit is a predefined upper limit of said Alpha values 

5. A method for simulating an appearance of products on an image of a consumer, as in claim 1, 
25 wherein 

said step of modifying appearance characteristics of said pixels of said product 
application area further comprises 



30 



- assigning hue and saturation appearance values of said pixels of said product 
application area equal to hue and saturation appearance values of said selected product 
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- assigning intensity appearance values of said pixels of said product application 
area according to intensity appearance values of a plurality of pixels within said product 
application area in said base image and according to intensity appearance values of a 
plurality of pixels outside said product application area 

5 

6. A method for simulating an appearance of products on an image of a consumer, as in claim 5, 
wherein; 

- said step of assigning intensity appearance values of said pixels of said product 
10 application area further comprises employing the following formula: 

Iout^A'Iin+B 

Where; 

15 lout is die value of the intensity of a pixel of the equalized area , 

lout can range between Upper and Lower Limits of 255 and 
0, respectively, 

Iin is the value of the intensity of an associated pixel of the product 
20 application area of the base image (before equalization), 

Iin can range between the Upper and Lower Limits, 

A is the slope of the function and is computed according to a gaussian 
distribution function, as follows: 

25 

A = exp(-0.5 * acolcrlmcan^/ttdev 2 ) 

Where, 

Icoicr is the value of the intensity of the applied 
30 product color, 

Imam is mean value of the intensities of the pixels in 
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the product application area, and 

Istdev is the variance of the intensities of the pixels in 

the product appEcation area, 



5 



Bis a constant computed by equating the !„ and lout values in the 



equalization function to I^n and Icoior , respectively. 

7. A method for simulating an appearance of products on an image of a consumer, as in claim 5, 
wherein: 



area further comprises assigning intensity appearance values of said pixels of said product 
application area according to intensity appearance values of substantially all of said pixels in said 
base image. 

15 8. A method for simulating an appearance of products on an image of a consumer, as in claim 7, 
wherein: 

- said step of assigning intensity appearance values of said pixels of said product 
application area further comprises employing the following formula: 



10 




>n 



20 



lout = A * Iin "KB 



Where; 

lout is the value of the intensity of a pixel of the equalized area , 



25 



lout can range between Upper and Lower Limits of 255 and 0, 
respectively, 



Iin is the value of the intensity of an associated pixel of the product 
application area of the base image (before equalization), 
Iin can range between the Upper and Lower Limits, 



30 



A is the slope of the function and is computed according to a gaussian 
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distribution function, as follows: 

A = exp(-0.5 * froicr W 2 /W) 
Where, 

5 Icote is the value of the intensity of the applied 

product 



10 



color, 

1^ is mean value of the intensities of the pixels in 
the product application area, and 
Istdcv is the variance of the intensities of the pixels in 
the product application area, 



B is a constant computed by equating the I* and W values in the 
15 equalization function to 1,^ and W , respectively. 

9. A method for simulating an appearance of products on an image of a consumer, comprising 

- compiling a database of a plurality of products, said database including appearance 
20 information for each said product 

- acquiring a base image of a consumer, said base image including a plurality of pixels 

- identifying a product application area in said base image, said product application area 
25 including a blending region, said product application area and said blending region thereof being 

defined by sets of pixels of said base image 

- assigning alpha channel values to each pixel of said set of pixels of said product 
application area, 

30 

- receiving a product selection from said consumer 
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- retrieving appearance information associated with said selected product from said 
database 

5 

- assigning hue and saturation appearance values of said pixels of said product application 
area substantially equal to hue and saturation appearance values of said selected product 

- assigning intensity appearance values of said pixels of said product application area 
10 according to intensity appearance values of a plurality of pixels within said product application 

area in said base image and according to intensity appearance values of substantially all of said 
pixels of said base image 

- said intensity are assigned according to a predetermined algorithm, comprising employing 
15 the following formula: 

Iout = A*Iin+B 

Where; 

20 I ont is the value of the intensity of a pixel of the equalized area , 

lout can range between Upper and Lower Limits of 255 and 
0, respectively, 

Im is the value of the intensity of an associated pixel of the product 
25 application area of the base image (before equalization), 

Iin can range between the Upper and Lower Limits, 



30 



A is the slope of the function and is computed according to a 
gaussian distribution function, as follows: 
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A = exp(-0.5 * acdcr-I^/Istdcv 2 ) 
Where, 

Icoicr is the value of the intensity of the applied 
product color, 

5 Imean is mean value of the intensities of the pixels in 

theproductapplicationarea,and 
Igdev is the variance of the intensities of the pixels in 
the product application area, 

10 

B is a constant computed by equating the I m and lout values in the 
equalization function to I^n and Icoi* , respectively. 

- blending pixels in said product application area of said product application area with said base 
15 image by modifying appearance information of said pixels of said blending region according to 

said alpha values 

- a red color value (Re) of said pixels of said product application area in said composite image 
being determined by the following formula: 

20 

Rc= [Rb * Alpha + Rp * (Alpha - UpperLimit)]/UpperLimit 
where 

Rb is a red color value of an associated pixel in said base image 
Rp is a red color value of said selected product; 
25 Alpha is an Alpha value of an associated pixel in said product application area; and 
UpperLiirrit is a predefined upper limit of said Alpha values 

a green color value (Gc) of said pixels of said product application area in said composite image 
being determined by the following formula: 

30 

Gc= [Gb * Alpha + Gp * (Alpha - UpperLimit)]/UpperLimit 
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where 

Gb is a green color value of an associated pixel in said base image 
Gp is a green color value of said selected product; 

Alpha is an Alpha value of an associated pixel in said product application area; and 

5 

UpperLinrit is a predefined upper limit of said Alpha values; and 

a blue color value (Be) of said pixels of said product application area in said composite image 
being determined by the following formula: 

10 

Bc= [Bb * Alpha + Bp * (Alpha - UpperLinA)J/UpperIimft 
where, 

Bb is a blue color value of an associated pixel in said base image 
Bp is a blue color value of said selected product; 
15 Alpha is an Alpha value of an associated pixel in said product application area; and 
UpperLimit is a predefined upper limit of said Alpha values 

- displaying a composite image of said base image as modified and blended in said product 
application area and said blending region thereof 

20 

- providing means to display an alternative product within said product application area 

10. A method for simulating an appearance of accessories on an image of consumer, comprising 

25 - compiling a database including an image of an accessory product and including an anchor point 
associated with said image of each said accessory product 

- acquiring a base image of a consumer, said base image including a plurality of pixels 
30 - defining an anchor point in said base image for location of an accessory product image 
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- calculating a base image size factor according to a pixel dimension of a body part of said 
consumer visible in said base image 

- calculating a base image rotation factor according to an orientation of said body part in said base 
5 image 

- receiving an accessory product selection from said consumer 

- retrieving a product image from said database associated with said accessory product selection 

10 

- adjusting a size of said product image according to said base image size factor 

- adjusting an orientation of said product image according to said base image rotation factor 

is - displaying a composite image of said base image with said product image superimposed 
thereover, said anchor point of said product image being substantially aligned with said anchor 
point of said base image 

- providing means to allow said consumer to adjust the size or orientation of said product image 

20 

- adjusting said size or orientation of said product image and displaying an adjusted composite 
image in response receiving associated instructions from said consumer. 

1 1 . A method for simulating an appearance of accessories on an image of a consumer, as in claim 
25 10 further comprising: 

- assigning alpha values for certain pixels of said product image 

- blending pixels of said product image having assigned alpha values with pixels of said base image 
30 according to said alpha values 
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12. A method for simulating an appearance of accessories on an image of a consumer, as in claim 
1 1, further comprising: 

- in said blending process, said alpha values resulting in certain areas of said product image being 
s substantially fully transparent and certain areas of said product image being semi-transparent 



13. A method of marketing aesthetic products, comprising: 

10 

- compiling a database of a plurality of aesthetic products, said database including appearance 
information for each said aesthetic product 

- acquiring a base image of a consumer, said base image including a plurality of pixels 

15 

- identifying a product application area in said base image, said product application area including 
a blending region, said product application area and said blending region thereof being defined by 
sets of pixels of said base image 

20 - selecting a product selection from said database 

- retrieving appearance information associated with said selected product from said database 

- modifying appearance information of said pixels of said product application area according to 
25 said appearance information of said selected product 

blending said product application area with said base image by modifying 
appearance information of said pixels of said blending region according to 
appearance 

30 - information of said selected product and 

according to appearance information of associated pixels of said base image in 
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said blending region 

- distributing, to said consumer, a composite image of said base image as modified and blended in 
said product application area and said blending region thereof 

5 

14. A method of marketing aesthetic products, as in claim 13 comprising: 

- distributing a sample of said selected product with said composite image 

10 IS. A method of marketing aesthetic products, as in claim 13, further comprising: 

- assigning alpha channel values to each pixel of said set of pixels of said product application area, 

- blending said product application area with said base image by modifying appearance 
15 characteristics of pixels of said product application area according to appearance information of 

said selected product and according to appearance information of associated pixels of said base 
image 

- in said blending step, weighing said appearance information of said associated pixels of said base 
20 image and appearance information of said selected product according to alpha values of associated 

pixels of said product application area 

16. A method of marketing aesthetic products, as in claim!5, further comprising: 

25 -in said blending step, weighing said appearance information of said associated pixels of said base 
image in reverse proportion to appearance information of said selected make-up product. 

17. A method of marketing aesthetic products, as in claim 16 further comprising: 



30 a red color value (Rc) of said pixels of said product application area in said composite image being 
determined by the following formula* 
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Ro= [Rb * Alpha + Rp * (Alpha - UpperLiraft)]/UpperLimit, where 
Rb is a red color value of an associated pixel in said base image 
Rp is a red color value of said selected product; 

Alpha is an Alpha value of an associated pixel in said product application area; and 
UpperLimit is a predefined upper limit of said Alpha values 

a green color value (Gc) of said pixels of said product application area in said composite image 
being determined by the following formula: 

Gc= [Gb * Alpha + Gp * (Alpha - UpperIimft)]/UpperLimit, where 
Gb is a green color value of an associated pixel in said base image 
Gp is a green color value of said selected product; 

Alpha is an Alpha value of an associated pixel in said product application area; and 
UpperLimit is a predefined upper limit of said Alpha values; and 

a blue color value (Be) of said pixels of said product application area in said composite image 
being determined by the following formula: 

Bc= [Bb * Alpha + Bp * (Alpha - UpperLimit)]/UpperLimit, where 
Bb is a blue color value of an associated pixel in said base image 
Bp is a blue color value of said selected product; 

Alpha is an Alpha value of an associated pixel in said product application area; and 
UpperLimit is a predefined upper limit of said Alpha values 

18. A method of marketing aesthetic products, as in claim 13, wherein: 

said step of modifying appearance characteristics of said pixels of said product application area 
further comprises 
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- assigning hue and saturation appearance values of said pixels of said product application area 
equal to hue and saturation appearance values of said selected product 



- assigning intensity appearance values of said pixels of said product application area according to 
intensity appearance values of a plurality of pixels within said product application area in said base 
image and according to intensity appearance values of a plurality of pixels outside said product 
application area. 

19. A method of marketing aesthetic products, as in claim 18, wherein: 



- said step of assigning intensity appearance values of said pixels of said product application area 
further comprises employing the following formula: 



Io* = A*Iin+B 



Where; 

Ioui is the value of the intensity of a pixel of the equalized area , 

lout can range between Upper and Lower Limits of 255 and 
0, respectively, 



lb is the value of the intensity of an associated pixel of the product 
application area of the base image (before equalization), 
Iin can range between the Upper and Lower Limits, 



A is the slope of the function and is computed according to a 
gaussian distribution function, as follows: 

A = exp(-0.5 * (Icolor-Imcan) 2 /Istdcv 2 ) 

Where, 

Icoior is the value of the intensity of the applied 
product color, 
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Imew is mean value of the intensities of the pixels in 
the product application area, and 
Istdev is the variance of the intensities of the pixels in 
the product application area, 

B is a constant computed by equating the Im and lout values in the 
equalization function to In»m and Icoior , respectively. 

20. A method of marketing aesthetic products, as in claim 1 8, wherein: 

said step of assigning intensity appearance values of said pixels of said product application area 
further comprises assigning intensity appearance values of said pixels of said product application 
area according to intensity appearance values of substantially all of said pixels in said base image. 

21. A method of marketing aesthetic products, as in claim 20, wherein: 

- said step of assigning intensity appearance values of said pixels of said product application area 
further comprises employing the following formula: 

Iout = A*Iin+B 

Where; 

lout is the value of the intensity of a pixel of the equalized area , 

lout can range between Upper and Lower Limits of 255 and 
0, respectively, 

Ik is the value of the intensity of an associated pixel of the product 
application area of the base image (before equalization), 
Iin can range between the Upper and Lower Limits, 

A is the slope of the function and is computed according to a 
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gaussian distribution function, as follows: 

A = exp(-0.5 * OcdorW 2 /!^ 2 ) 
\ 

5 ] Where, 

Icoior is the value of the intensity of the applied 
product color, 

Imam is mean value of the intensities of the pixels in 
the product application area, and 
10 Istdcv is the variance of the intensities of the pixels in 

the product application area, 
B is a constant computed by equating the I m and la* values in the 
equalization function to Imean and Icoior , respectively. 

15 22. A method for simulating an appearance of products on an image of a consumer as in claim 1, 
further comprising: 

interconnecting first and second user systems through a network; 

20 receiving a product selection from a first user, 

displaying a composite image incorporating said product selected by said first user on said first 
and second user systems. 
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FIG. 5 
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FIG. 6 
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FIG. 7 
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FIG. 8 
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FIG. 9 
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FIG. 10 
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FIG. 11 
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FIG. 12 
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FIG. 13 




13/25 



SUBSTITUTE SHEET (RULE 26) 



WO 02/05249 



PCT/IB01/01495 



FIG. 14 
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FIG. 15 
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FIG. 16 
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FIG. 17 




17/25 



SUBSTITUTE SHEET (RULE 26) 



WO 02/05249 



PCT/IB01/01495 



FIG. 18 
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FIG. 21 
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FIG. 22 
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FIG. 23 
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FIG. 24 
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FIG. 25 
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